haskell.compiler.ghc*: cross-compiled ghc [iteration on alex's PR] #305392

sternenseemann · 2024-04-19T20:44:34Z

Description of changes

Things done

Add a 👍 reaction to pull requests you find important.

AlexandreTunstall · 2024-04-19T21:20:13Z

I like the change to use ghc-settings-edit, but would you mind adding a comment inside the LHS file and/or the meta description to briefly explain what the script is used for?

As mentioned in this comment of the original PR, the settings file apparently still references the build platform's LLVM.
Have you had a chance to look into this issue?
I assume it just needs an extra rule to overwrite the LLVM field.

I'll test pkgsCross.riscv64 and report back in a few days.

sternenseemann · 2024-04-19T21:40:36Z

but would you mind adding a comment inside the LHS file and/or the meta description to briefly explain what the script is used for?

Yes, I'll do that.

still references the the build platform's LLVM

Forgot about that, I'll look into that. This is basically the case the old asserts guarded against?

assert targetCC == pkgsHostTarget.targetPackages.stdenv.cc;
assert buildTargetLlvmPackages.llvm == llvmPackages.llvm;

AlexandreTunstall · 2024-04-19T21:49:38Z

This is basically the case the old asserts guarded against?

I think so.

AlexandreTunstall · 2024-04-23T03:01:59Z

I have successfully used this PR's branch to boot a native compiler and build nix-output-monitor and ShellCheck on RISC-V.

The GHC 9.4.8 cross-build still needs to be unregisterised, but through the rebuild I was able to obtain a fully functional native RISC-V compiler, so there has been no regression on that front.

Since a cross-compiled 9.4.8 can be used to boot a native 9.6.5, please feel free to update the cross sanity test to build 9.4.8 instead of 8.10.7.
I can't remember why my past testing led me to believe that only 8.10.7 was usable as a boot compiler, but it no longer seems to be an issue.

AlexandreTunstall · 2024-04-23T16:02:05Z

This error is probably what led me to believe that only 8.10 and 9.0 were usable as boot compilers:

ghc.mk:141: *** The make build system requires a boot compiler older than ghc-9.2. Your boot compiler is too new and cannot be used to build ghc-9.4 with make. Either boot with ghc 9.0.2 or build with hadrian. See https://www.haskell.org/ghc/blog/20220805-make-to-hadrian.html for advice on transitioning to hadrian.. Stop.

At the time, Nixpkgs had not yet switched to 9.6.

sternenseemann · 2024-04-27T22:36:46Z

Added some commentary to the ghc-settings-edit.lhs file.

sternenseemann · 2024-06-03T23:36:26Z

port everything to ghc-settings-edit
cross splicing for libdw
cross splicing for gmp
fix unlit in settings (ref https://gitlab.haskell.org/ghc/ghc/-/issues/23317)
fix LLVM tools in settings

Follow-up PRs:

haskell.compiler.ghc*: use matching CLANG & CC on Darwin with LLVM #338782
rename buildCC, targetCC, installCC
extend release-cross.nix
extend release-haskell.nix

sternenseemann · 2024-09-05T21:47:27Z

@AlexandreTunstall Does enableDocs imply enableHaddock? Is that why you made the condition for the former more strict?

sternenseemann · 2024-09-07T12:03:42Z

/nix/store/6pmlfcq95bswq8vwb64wzasp9pk8pnm2-ghc-9.8.2
/nix/store/fqmmy8qahfh7yhpm9v4xfddfw56w8y2a-ghc-9.8.2-doc
/nix/store/zg2lzsx6ni1s6lm0wvznf2bl1lasii9v-ghc-aarch64-unknown-linux-gnu-8.10.7-doc
/nix/store/znjbb57b6jfq1kqcg67rcw4wvv3yszn9-ghc-aarch64-unknown-linux-gnu-8.10.7
/nix/store/7cmg5rfmnr07mms8fami2qkwbbfpswpv-aarch64-unknown-linux-gnu-ghc-9.6.6-doc
/nix/store/cam5dzipi1yq3y9nfx64mwr8vm6nhlkk-aarch64-unknown-linux-gnu-ghc-9.6.6
/nix/store/8pg837wpi3k6x82i715bkj6hg072ngv8-ghc-aarch64-unknown-linux-gnu-9.4.8-doc
/nix/store/wg0mppma2z6spmj6np32kzk2xh1qlrh9-ghc-aarch64-unknown-linux-gnu-9.4.8

Successfully compile.

GHC's build system assumes that the C compiler, tools etc. discovered during configure can also be used at runtime. This means that the CC, LD, AR etc. variables given at runtime are used to populate the settings file which GHC uses to lookup the tools it needs. The implicit assumption of this mechanism is that the build and runtime environment of GHC are similar enough and PATH is used to find the tools. I. e. if we set CC=clang, we wouldn't need to worry about this as much. We, however, pass absolute paths which is useful since it allows GHC to work outside of stdenv (as long as e. g. no FFI is involved). Even so, until now, we didn't really have any problems stemming from this, as we used pkgsBuildTarget to get everything we need. The compiler we'd want to execute would in principle need to come from pkgsHostTarget. 1. For native compilers, all package sets are the same since build == host == target. 2. For cross compilers build == host, so pkgsBuildTarget is practically the same as pkgsHostTarget. When cross-compiling a native compiler, build != host, so we need to actually ensure that GHC uses different tools at runtime compared to bootstrapping. There is currently no intended way to achieve this, so we use a custom tool to edit the settings file. An alternative would be to patch the build system, but this would be difficult to maintain. We could go down this route if there's interest from upstream to provide a proper way to specify the runtime tools. Co-authored-by: sternenseemann <[email protected]>

sternenseemann · 2024-09-07T13:18:31Z

Successfully (natively) compiled test programs with GHC 8.10.7, 9.4.8 cross compiled from x86_64-linux on aarch64-linux.
“Ordinary” Cross compilation from x86_64 to aarch64-linux with 9.6.6 still works.
Native 9.8.2 compiler still works.

AlexandreTunstall · 2024-09-07T13:19:04Z

Does enableDocs imply enableHaddock? Is that why you made the condition for the former more strict?

It's been a while since I wrote that change, so I can't remember.
One would have to try enableDocs = true; enableHaddock = false and see, because I can't find anything in the GHC documentation that suggests that one implies the other.

AlexandreTunstall · 2024-09-07T14:16:06Z

pkgs/test/cross/default.nix

@@ -146,6 +146,8 @@ let
    pkgs.pkgsCross.mips64el-linux-gnuabi64.stdenv
    pkgs.pkgsCross.mips64el-linux-gnuabin32.stdenv
    pkgs.pkgsCross.mingwW64.stdenv
+    # Unlike later versions, cross-compiled 8.10.7 can reliably bootstrap a native build
+    pkgs.pkgsCross.riscv64.haskell.compiler.integer-simple.ghc8107


Should we update this to 9.4.8 now that we know it works on aarch64 and riscv64?

(I originally used 8.10.7 because an unrelated problem prevented me from bootstrapping with a newer compiler.)

Maybe an idea, I think we can expand the test list a bit (in release-haskell.nix), but should keep it to the essential stuff in the test.

The nice thing about 8.10.7 is of course that it can be used to start a bootstrap chain of all compilers in nixpkgs.

Also, we should be able to use gmp now if gmp can be cross compiled to riscv64.

AlexandreTunstall · 2024-09-07T16:57:04Z

On the latest version of this PR, pkgsCross.riscv64.haskell.compiler.native-bignum.ghc948 builds, runs natively on my RV64GC_Zba_Zbb system, and outputs working RV64GC binaries.

sternenseemann · 2024-09-08T19:55:55Z

One would have to try enableDocs = true; enableHaddock = false and see, because I can't find anything in the GHC documentation that suggests that one implies the other.

I've confirmed that it only influences the manual build which isn't problematic when cross-compiling GHC. We only need/should disable it if it is a hassle to build sphinx for the host platform (e.g. as is the case with musl).

Propagating the supported platforms of the boot compiler doesn't make much sense when unregistered cross-compilation is possible.

This is to ensure that Haskell users on platforms that lack official bindists still have a convenient means of getting GHC running natively. In my admittedly somewhat limited testing on RISC-V, GHC 8.10.7 is able to bootstrap native builds for 9.2.8 and 9.4.5. GHC 9.2.8 and 9.4.5 are unable to bootstrap themselves and 9.6.2 when cross-compiled. If you're looking at this commit to see whether you can safely upgrade the compiler used here to remove 8.10, please try cross-compiling 9.0 or later and then booting a native GHC with it.

Hadrian started installing unlit with a targetPrefix (if applicable) which wasn't the case with make before. Unfortunately, the logic to generate the settings file wasn't updated, so GHC 9.6.* cross compilers expect to find an unlit binary without a target prefix. Upstream issue: https://gitlab.haskell.org/ghc/ghc/-/issues/23317

The build platform doesn't matter for checking which stage is our final stage! Stage2 means host and target are sufficiently similar, Stage1 means host and target may differ.

Note that the clang command should be adapted in the future as done for CLANG in NixOS#338782.

github-actions bot added the 6.topic: haskell label Apr 19, 2024

ofborg bot added the 6.topic: cross-compilation Building packages on a different platform than they will be used on label Apr 19, 2024

ofborg bot added the 8.has: package (new) This PR adds a new package label Apr 19, 2024

ofborg bot requested review from cdepillabout, expipiplus1, ncfavier, guibou and maralorn April 19, 2024 21:36

ofborg bot added 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Apr 19, 2024

sternenseemann force-pushed the ghc-cross branch from da72955 to c5f0619 Compare April 27, 2024 22:35

sternenseemann force-pushed the ghc-cross branch from c5f0619 to cb0e5d0 Compare April 28, 2024 08:49

sternenseemann force-pushed the ghc-cross branch 3 times, most recently from 1e7476f to 39fbcaf Compare May 9, 2024 20:23

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label May 22, 2024

sternenseemann force-pushed the ghc-cross branch from 39fbcaf to fedb5a4 Compare June 3, 2024 23:34

sternenseemann removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Jun 3, 2024

sternenseemann force-pushed the ghc-cross branch from fedb5a4 to 75cab63 Compare June 11, 2024 08:40

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 6, 2024

sternenseemann force-pushed the ghc-cross branch 2 times, most recently from ca53372 to 3ff0c66 Compare September 5, 2024 20:41

sternenseemann force-pushed the ghc-cross branch 2 times, most recently from f8dfa10 to 15afc8b Compare September 6, 2024 21:30

sternenseemann and others added 2 commits September 7, 2024 14:05

ghc-settings-edit: init at 0.1.0

dd1c4a7

sternenseemann force-pushed the ghc-cross branch from 15afc8b to 9e3f639 Compare September 7, 2024 12:05

sternenseemann marked this pull request as ready for review September 7, 2024 13:18

AlexandreTunstall reviewed Sep 7, 2024

View reviewed changes

sternenseemann force-pushed the ghc-cross branch from 9e3f639 to 6957b21 Compare September 8, 2024 18:11

AlexandreTunstall and others added 10 commits September 8, 2024 22:18

haskell.compiler.ghc*: don't cross-compile haddock

9ac88ee

haskell.compiler.ghc*: add enableUnregisterised

c1dbadc

haskell.compiler.ghc*: assert cross is possible

55b6cf6

haskell.compiler.ghc*: allow all platforms

30883fd

Propagating the supported platforms of the boot compiler doesn't make much sense when unregistered cross-compilation is possible.

haskell.compiler.ghc*: use GHC from pkgsBuildBuild

9351f3e

haskell.compiler.ghc9{6,8,10}: update comment about stage selection

a7d8b93

The build platform doesn't matter for checking which stage is our final stage! Stage2 means host and target are sufficiently similar, Stage1 means host and target may differ.

haskell.compiler.*: set cctools tools in settings if applicable

6e07628

haskell.compiler.*: set correct runtime LLVM tools in settings

df16902

Note that the clang command should be adapted in the future as done for CLANG in NixOS#338782.

sternenseemann force-pushed the ghc-cross branch from 6957b21 to df16902 Compare September 8, 2024 20:21

sternenseemann merged commit 2dae227 into NixOS:haskell-updates Sep 8, 2024
22 of 26 checks passed

sternenseemann deleted the ghc-cross branch September 8, 2024 21:50

sternenseemann mentioned this pull request Sep 19, 2024

haskell.compiler.ghc*: fix cross-built native GHC #243619

Closed

25 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

haskell.compiler.ghc*: cross-compiled ghc [iteration on alex's PR] #305392

haskell.compiler.ghc*: cross-compiled ghc [iteration on alex's PR] #305392

sternenseemann commented Apr 19, 2024

AlexandreTunstall commented Apr 19, 2024 •

edited

Loading

sternenseemann commented Apr 19, 2024

AlexandreTunstall commented Apr 19, 2024

AlexandreTunstall commented Apr 23, 2024

AlexandreTunstall commented Apr 23, 2024

sternenseemann commented Apr 27, 2024

sternenseemann commented Jun 3, 2024 •

edited

Loading

sternenseemann commented Sep 5, 2024

sternenseemann commented Sep 7, 2024

sternenseemann commented Sep 7, 2024

AlexandreTunstall commented Sep 7, 2024

AlexandreTunstall Sep 7, 2024

sternenseemann Sep 8, 2024

AlexandreTunstall commented Sep 7, 2024

sternenseemann commented Sep 8, 2024

haskell.compiler.ghc*: cross-compiled ghc [iteration on alex's PR] #305392

haskell.compiler.ghc*: cross-compiled ghc [iteration on alex's PR] #305392

Conversation

sternenseemann commented Apr 19, 2024

Description of changes

Things done

AlexandreTunstall commented Apr 19, 2024 • edited Loading

sternenseemann commented Apr 19, 2024

AlexandreTunstall commented Apr 19, 2024

AlexandreTunstall commented Apr 23, 2024

AlexandreTunstall commented Apr 23, 2024

sternenseemann commented Apr 27, 2024

sternenseemann commented Jun 3, 2024 • edited Loading

sternenseemann commented Sep 5, 2024

sternenseemann commented Sep 7, 2024

sternenseemann commented Sep 7, 2024

AlexandreTunstall commented Sep 7, 2024

AlexandreTunstall Sep 7, 2024

Choose a reason for hiding this comment

sternenseemann Sep 8, 2024

Choose a reason for hiding this comment

AlexandreTunstall commented Sep 7, 2024

sternenseemann commented Sep 8, 2024

AlexandreTunstall commented Apr 19, 2024 •

edited

Loading

sternenseemann commented Jun 3, 2024 •

edited

Loading